How I brought down build times from 25 minutes to under 2, and cut image sizes by 95%

TL;DR

This is a real-world story of how I optimized the Docker build and deployment pipeline for a Golang-based service:

• Reduced build time from ~25 minutes to under 2

• Dropped image size from 2GB+ to just ~80MB

• Improved Kubernetes pod startup and hotfix deployment speed

• Cut ECR storage costs and made CI pipelines faster & more reliable

• Shifted to secure, minimal base images with stripped, statically compiled Go binaries

• Enabled BuildKit + remote caching via ECR for cache-friendly, repeatable builds

• Standardized reusable Docker patterns across multiple services

💬 Heads-up: This is a long one — but if you're building Go apps for production, it's probably the only guide you’ll ever need to run them in containers efficiently, securely, and at scale.

Let’s be honest, no one likes waiting 25 minutes for a Docker image to build, especially when you're in the middle of development and just want to test a small change or worse, when your production system is down and you need to push a quick fix, fast.

That was exactly the situation I found myself in.

We had Golang-based microservices that worked well, but everything around them was painfully slow. Docker builds took forever. The image sizes were close to 2GB. And our CI/CD pipeline felt sluggish - every build and deploy step added unnecessary delays. On top of that, Kubernetes pod and ECS task startups were slow, and our ECR storage costs kept going up.

At first, it seemed like a few minor issues. But once I dug in, I found all the usual suspects: outdated Dockerfile, no caching, heavy base images, and no attention to what was actually getting packed into the final container.

This blog is a real-world story of how I brought down Docker build times from 25 minutes to under 2, reduced image size by 95%, made deployments faster, and saved on storage costs - all with small, focused changes that had a big impact.

Let me walk you through how it all happened.

The Setup: When Things Took Forever

When I first looked at the repos, everything seemed fine on the surface. The Go codebases were clean, the microservices worked well, and the functionalities were solid. But the moment I triggered a build, I knew something wasn’t right.

• Build time: 20 to 25 minutes

• Docker image size: ~2GB

• CI/CD: painfully slow

• ECR storage: growing steadily

• Kubernetes pod and ECS task spin-up: delayed every time due to heavy image pulls

Whether the team was pushing a regular feature or trying to ship a hotfix under pressure, it felt like the pipeline was working against us. And it wasn’t just about time - it was mentally exhausting. Waiting 25 minutes just to test a minor change kills momentum, and during production incidents, it made things worse.

We were using AWS EKS and AWS ECS to deploy services, and naturally, the large image size made everything downstream slow too - from image pulls to container starts. Over time, the impact started showing up in our cloud bills as well, thanks to the bloated ECR usage.

That’s when I decided to dig deeper and figure out what was really going on inside the Dockerfiles.

Legacy State: What Was Wrong

Once I dug deep into the Dockerfiles, it became pretty clear why things were so slow.

It had all the classic red flags of a first-draft or "it-just-works" setup - something likely written in a hurry or by someone new to Docker best practices (we’ve all been there).

Here’s what I found:

1. Heavy base images

   The build used a full-fledged golang:1.xx image, and sometimes even an ubuntu base layer stacked below it - which pulled in hundreds of MBs of unnecessary tools and packages.

2. Improper multi-stage build

   Technically, there were multiple stages defined - but they weren’t used the right way.
   In the end, everything - including build tools, test binaries, source files, and Go module cache - was copied into the final stage anyway. The whole point of isolating build and runtime environments was lost, which made the final image just as bloated as a single-stage build.

3. No .dockerignore

   Every time the image was built, it copied the entire repo - including .git, local configs, test files, and sometimes even large assets. That bloated the build context and slowed down the docker build process.

4. Dependencies rebuilt every time

   The build didn’t cache go mod download or any dependencies, so even the tiniest code change triggered a full re-download of all modules. This added several minutes to every build and made CI unnecessarily slow.

5. Too many layers, unoptimized instructions

   Each RUN, COPY, and ADD instruction created a new image layer. And many of them were repetitive or could have been merged to reduce overhead. No layer caching strategy meant the whole image rebuilt far too often.

All of this combined made the image big, the build slow, and the pipeline fragile. And because it wasn’t modular or maintainable, any optimization felt risky - which is often why these setups stay broken for so long. But with just a few focused improvements, I knew we could fix all of it.

Starting the Fix: Where I Began

Once I had a clear picture of what was going wrong, I didn’t try to solve everything at once. I started with the low-hanging fruit - small changes that I knew would give immediate results without breaking anything.

1. Add a .dockerignore

   This was the simplest win. Until then, our Docker builds were copying everything - including .git, test files, README docs, local configs, and even random developer junk lying around.

   By adding a proper .dockerignore file, it reduced the build context dramatically. This alone cut ~15–20 seconds from every build and reduced noise inside the image.

    # .dockerignore
    .git
    *.md
    test/
    *.log
    *.local
    node_modules/
   

2. Clean up and restructure the Dockerfile

   Next, I reorganized the Dockerfile to follow a proper multi-stage structure.

   Earlier, we had multiple stages defined - but the final stage copied almost everything again, defeating the purpose. I fixed that by making sure the final image only contained the stripped, statically compiled Go binary, nothing else.

   This also meant removing all build tools, C dependencies, and temp files from the final image - instantly shrinking the size.

3. Build a custom base image for heavy dependencies

   One of the biggest time sinks in our builds was installing C libraries like librdkafka, libssl, and other native packages - every single time.

   To solve this, I created a custom Docker base image that included all these heavy dependencies pre-installed. This way, our actual app builds could start from a pre-baked image that didn’t need to reinstall or download anything at runtime.

   It took some upfront effort, but the payoff was huge:

   • Reduced build time by several minutes

   • Made the builds more stable and reproducible

   • Allowed us to reuse the base image across multiple Go services

4. Switch to a minimal final base image

   Instead of shipping everything inside a golang or ubuntu image, I switched to using scratch or distroless/static as the base for the final stage.

   This change alone dropped our final image from over 2GB to ~80MB. It also made the image more secure by removing shells, compilers, and unnecessary binaries.

5. Enable Go build caching

   Previously, our builds were re-downloading modules and rebuilding everything from scratch. To fix this, I added Docker cache mounts to persist Go’s build and module cache between builds:

    ENV GOCACHE=/root/.cache/go-build
    RUN --mount=type=cache,target=/root/.cache/go-build \
        go build -o app .
   

   How this works:

   • GOCACHE tells Go where to store its compiled build artifacts (object files, dependency build outputs, etc.)

   • The --mount=type=cache,target=... directive is a special Docker BuildKit feature that preserves that cache between builds

   • This cache is persisted outside the image layers, so Docker can reuse it even when the source changes slightly

   • On subsequent builds, Go can detect unchanged packages and skip rebuilding them entirely - drastically cutting down build time

This made a huge difference - especially in CI pipelines where minor code changes were triggering full builds earlier. Now, unchanged dependencies didn’t add to build time anymore.

6. Strip the binary and build statically

   To make the binary as lean as possible and compatible with minimal base images like scratch, I added flags to:

   • Strip debug info and symbols

   • Ensure it was statically compiled (very important when using native C dependencies like librdkafka)

       CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
       go build -ldflags="-s -w" -o app .
     

     • CGO_ENABLED=0 ensures no dynamic linking

     • -ldflags="-s -w" strips debugging symbols for smaller binaries

⚠️ Note: In cases where native C libraries like librdkafka are actually needed at runtime, then ensure CGO_ENABLED=1 and use a custom base image to handle linking cleanly.

7. Configure Docker BuildKit in CI/CD

   Most of these optimizations rely on Docker BuildKit, so I made sure it was explicitly enabled in the CI pipeline.

   For example, in GitLab CI:

    variables:
      DOCKER_BUILDKIT: 1
      BUILDKIT_INLINE_CACHE: 1
   

   This unlocked:

   • Inline cache support

   • Parallel layer execution

   • Cache mounts for Go build cache

Without this, none of the caching improvements would work consistently in CI/CD.

8. Remote layer caching using ECR

   To make builds even faster in our GitLab CI runners, I set up remote cache pushing to Amazon ECR. This allowed subsequent pipeline runs to pull existing layers instead of rebuilding everything from scratch - even across branches and commits.

    docker build \
      --cache-from=type=registry,ref=<account>.dkr.ecr.region.amazonaws.com/my-app:buildcache \
      --cache-to=type=registry,mode=max,ref=<account>.dkr.ecr.region.amazonaws.com/my-app:buildcache \
      -t my-app:latest .
   

   This worked especially well when builds ran on ephemeral runners (k8s runners), where local layer caching wasn’t persistent. By using ECR as a remote cache store, I preserved all heavy layers like base images, Go modules, and C library installs - and skipped them on every incremental build.

Iterating & Fine-Tuning

Once the big issues were out of the way - oversized base images, missing .dockerignore, and cache-less builds - things were already much better. But I didn’t stop there.

This is where I went a bit deeper - looking at what else could be optimized to squeeze out every second and make the setup truly production-grade.

1. Reordered Dockerfile layers for better cache hits

   In Docker, the order of instructions matters a lot when it comes to caching. I made sure that:

   • COPY go.mod and COPY go.sum came before the full source code

   • go mod download was in its own layer

This ensured that if I didn’t change dependencies, Docker would reuse that layer and skip re-downloading modules.

    COPY go.mod go.sum ./
    RUN go mod download

    COPY . .
    RUN go build -o app .

This simple reorder reduced 30–60 seconds on average per build in CI.

2. Removed unused packages and binaries

   Once I moved to a custom base image and clean multi-stage builds, I ran docker history (very handy command) on the final image to inspect what was still there - and found some hidden leftovers.

   I removed:

   • Unused tools (like curl, jq, etc. mistakenly left from early setups)

   • Redundant COPY steps

   • Old static files that weren’t used at runtime

Every MB mattered - especially in environments like EKS and ECS, where smaller images meant faster pod spinups and fewer image pull throttles.

3. Split test/lint/build stages in CI

   To speed up feedback in CI pipelines, I split the stages:

   • Run lint + tests first (fast fail if needed)

   • Build and push image only if tests pass

This avoided wasting time building/pushing a container if tests were going to fail anyway.

4. Tagged and reused base layers across services

   Since I had multiple Go services with similar dependencies (Kafka, Redis, etc.), I reused the same custom base image across them.

   This made:

   • Build times more predictable

   • ECR usage more efficient

   • CI logs less noisy (Docker could reuse layers)

These refinements weren’t dramatic on their own, but together they added polish and consistency. The whole system felt lighter, faster, and more maintainable - and most importantly, everyone in the team could feel the difference.

Results: What We Gained

Once all the optimizations were in place, the impact was immediately visible - not just in numbers, but in the overall developer experience, CI speed, and even how fast things shipped to production.

Here’s what we achieved:

There were multiple incidents where something broke in prod - and thanks to the optimized build pipeline and smaller images, we were able to patch the issues, run tests and build & deploy to production - all within a few minutes

This directly helped us maintain uptime and stay on top of incident SLAs - something that would have been impossible with the old 25-minute build pipeline.

When things go wrong, speed matters. And this speed gave us control, not chaos.

Final Thoughts and Reflections

Looking back, this wasn’t about chasing build speed for the sake of it. It was about removing friction - the kind that silently slows down shipping, wears down developers, and becomes a hidden tax on every deploy.

The original setup wasn’t “wrong” - it was just what happens when things are built fast, without time to reflect on best practices. It was the natural outcome of time constraints, unclear ownership, and the all-too-common “it works for now” mindset. But once I started cleaning it up - step by step, the benefits were massive… especially during a production outage.

It wasn’t magic. Just:

• Respecting Docker’s caching model

• Keeping the final image as lean as possible

• Building with clarity and purpose

What really made the difference was treating Docker as an active part of the development experience, not just a deployment afterthought.

💡 Tips for Teams Starting Out

If you're building Go apps and containerizing them for production, here are a few things I wish we had nailed from day one:

• Don’t ignore Docker best practices: The defaults usually “work”, but they rarely work well. Spend time structuring your Dockerfile properly, it pays back quickly.

• Bake in security and reproducibility from day one: Use minimal base images, strip sensitive tools, and lock down environments early. Avoid surprises later.

• Use multi-stage builds religiously: Never ship compilers, tools, or test data in your final image. It’s not just about size, it’s about safety and clarity.

• Static Go binaries are a gift - use them well: CGO_ENABLED=0 + scratch or distroless can take you a long way. Simpler, safer, smaller.

• Enable BuildKit early: It's not optional anymore - it unlocks all the caching and performance improvements you need for modern CI/CD.

And when you ensure these things, your future self (and your team) will thank you :)

In the world of software engineering, proficiency in Linux and command-line tools is essential. Whether you're managing servers, deploying applications, or debugging code, a solid grasp of Linux and shell scripting empowers you to navigate the intricate landscape of software development. This comprehensive guide aims to equip you with the fundamental knowledge required to harness the power of Linux and effectively utilize the command line.

From understanding the basic concepts of Linux distributions and file systems to mastering essential commands for navigation, configuring networks, writing shell scripts, and navigating Linux Man pages, this guide covers a wide range of topics. Whether you're a beginner stepping into the world of Linux or a seasoned developer looking to enhance your command-line skills, this guide is tailored to meet your needs.

Throughout this guide, we'll explore key concepts, provide clear explanations, offer practical examples, and share useful resources that will accelerate your proficiency in Linux and shell scripting. By the end of this guide, you'll have a strong foundation that enables you to confidently work within Linux environments, efficiently manage tasks, automate processes, and troubleshoot issues.

Ready to dive in? Let's get started!

1. What is Linux?

Overview of Linux

Linux is an open-source operating system kernel developed by Linus Torvalds in the early 1990s. It differs from proprietary operating systems by allowing users to access and modify its source code.

Linux Distributions and Their Role

Linux distributions combine the Linux kernel with software packages to create complete operating systems. Distributions cater to specific use cases, such as Ubuntu for desktops and CentOS for servers.

Linux Features for Software Development

Linux provides an environment conducive to software engineering, supporting various programming languages, development tools, and frameworks. The command-line interface (CLI) allows developers to interact with the system directly.

2. Introduction to the Shell

Definition of the Shell and Its Importance

The Shell is a command-line interface that facilitates interaction with the operating system. It processes user commands and communicates with the kernel to execute those commands.

Types of Shells

Different Shells, such as bash, sh, and zsh, offer varying features. Bash, the Bourne-Again Shell, is widely used and comes pre-installed on many Linux distributions.

Accessing the Shell on Linux

To access the Shell, open the terminal application on your Linux system. The terminal provides a text-based interface for entering commands.

3. Navigating the File System

Understanding the File System Hierarchy

Linux follows a hierarchical file system structure with the root directory denoted as '/'. All files and directories are organized beneath it.

Key Directories and Their Purposes

The Linux file system hierarchy is organized in a structured manner, with various directories serving specific roles. Understanding these directories is essential for effective navigation and management of the system.

Basic Commands for Navigation

These basic commands are essential for navigating and interacting with the file system in Linux. Familiarizing yourself with them is a fundamental step toward effective command-line usage.

4. File and Directory Manipulation

Creating, Copying, Moving, and Deleting

• touch: Create empty files

• cp: Copy files and directories

• mv: Move or rename files and directories

• rm: Remove files and directories

Combining Commands Using Pipes and Redirection

Pipes (|) direct output from one command to another, while redirection (>, >>) sends output to files.

5. Text Processing

Using Commands for Text Manipulation

• cat: Display file contents

• grep: Search for patterns in files

• sed: Stream editor for text manipulation

Understanding Regular Expressions

Regular expressions (regex) are patterns used for text matching and manipulation. They are powerful tools for complex text operations.

For a more comprehensive understanding, explore the following resource: https://www3.ntu.edu.sg/home/ehchua/programming/howto/Regexe.html

6. File Permissions and Users

Explanation of File Permissions

In Linux, file permissions control who can access, modify, or execute files and directories. These permissions ensure data security and system integrity by restricting unauthorized access.

Permissions are categorized into three groups: user (u), group (g), and others (a). Each group has three types of permissions: read (r), write (w), and execute (x). The presence or absence of these permissions determines what actions users can perform on a file or directory.

Understanding Permission Modes (u/g/a)

Permission modes consist of combinations of rwx permissions for the user, group, and others. They are represented as a three-digit number, where each digit corresponds to a permission group. For instance, the permission mode "644" corresponds to read and write permissions for the user (6), and read-only permissions for the group and others (4).

• r (read): Allows viewing the content of a file or listing directory contents.

• w (write): Permits modifying a file's content or creating/deleting files in a directory.

• x (execute): Grants the ability to run executable files or access a directory's contents.

Binary Permissions (e.g., 777, 660)

Binary permissions are another representation of permission modes, using a three-digit binary number. Each digit corresponds to the presence (1) or absence (0) of read, write, or execute permissions. The three digits represent user, group, and others in that order.

• r is represented by 4 (100 in binary)

• w is represented by 2 (010 in binary)

• x is represented by 1 (001 in binary)

For example, the permission mode "777" signifies read, write, and execute permissions for user, group, and others, resulting in full access. Similarly, "660" grants read and write permissions for user and group, while denying access to others.

Setting and Modifying Permissions

To set permissions, use the chmod command followed by the permission mode and the target file/directory. For instance, chmod 755 file.txt grants read, write, and execute permissions to the user, and read and execute permissions to the group and others.

Changing Ownership

The chown command allows changing file ownership. Users can assign files to themselves or other users, and also change group ownership using the chgrp command. File permissions are essential for securing your files and ensuring proper access control within a Linux system.

7. Package Management

Introduction to Package Managers

Package managers are crucial tools for managing software installation, updates, and removal on Linux systems. Different Linux distributions come with their own package managers, each tailored to the distribution's design and requirements. Here are some major Linux distributions and their associated package managers:

Debian-based Distributions (e.g., Ubuntu, Debian)

• APT (Advanced Package Tool): APT is the default package manager for Debian-based distributions. It offers a user-friendly command-line interface for managing software. Common APT commands include:

  • apt-get update: Update package information

  • apt-get install <package>: Install a package

  • apt-get upgrade: Upgrade installed packages

Red Hat-based Distributions (e.g., CentOS, Fedora)

• YUM (Yellowdog Updater Modified): YUM is the package manager used primarily in Red Hat-based distributions. It simplifies package management tasks and resolves dependencies automatically. Some YUM commands are:

  • yum update: Update all packages

  • yum install <package>: Install a package

  • yum remove <package>: Remove a package

Arch Linux

• Pacman (Package Manager): Pacman is the package manager for Arch Linux and its derivatives. Known for its simplicity and speed, Pacman commands include:

  • pacman -Syu: Update package database and upgrade system

  • pacman -S <package>: Install a package

  • pacman -R <package>: Remove a package

SUSE-based Distributions (e.g., openSUSE)

• ZYpp (Zypper): Zypper is the package manager used in SUSE Linux distributions. It offers both command-line and graphical interfaces for package management. Basic Zypper commands include:

  • zypper refresh: Refresh repository metadata

  • zypper install <package>: Install a package

  • zypper remove <package>: Remove a package

Managing Software Packages

Understanding your distribution's package manager is crucial for maintaining a stable and up-to-date system. Different distributions have their own package repositories, ensuring you have access to a wide range of software tailored for your system.

8. Process Management

Monitoring and Controlling Processes

• ps: Display running processes

• top: Monitor system processes

• kill: Terminate processes

Running Processes in the Background and Foreground

Use & to run a process in the background and fg to bring a background process to the foreground.

9. File Editing

Using Text Editors

Text editors like nano and vim allow you to create and edit files from the terminal.

Basic Editing Commands and Navigation

Text editors have specific keybindings for navigation, editing, and saving files.

10. Networking Basics

Configuring Network Settings

Networking is a crucial aspect of system administration, enabling communication between devices. Linux provides tools to configure network settings effectively:

Using Tools for Network Troubleshooting

Troubleshooting network issues is essential to maintain a smooth operation. Linux offers several tools for diagnosing and resolving connectivity problems:

Understanding these networking basics and utilizing the provided commands can greatly assist in configuring, maintaining, and troubleshooting network connectivity on Linux systems.

11. Shell Scripting

Introduction to Shell Scripting

Shell scripting is a powerful way to automate tasks in Linux. It involves writing a series of commands that are executed in a sequence, similar to a program. Shell scripts are particularly useful for repetitive tasks, system administration, and batch processing.

Writing Your First Shell Script

To create a simple shell script, follow these steps:

1. Open a text editor (such as nano, vim, or a GUI editor).

2. Begin the script with a shebang line, which specifies the interpreter. For example: #!/bin/bash indicates that the script should be interpreted by the Bash shell.

3. Write your script commands below the shebang line.

Example: Hello World Script

Here's a basic "Hello, World!" shell script hello.sh:

#!/bin/bash 
echo "Hello, World!"

To run the script:

1. Open a terminal.

2. Navigate to the directory containing hello.sh using cd.

3. Provide execute permission to the user: chmod +x hello.sh.

4. Run the script: ./hello.sh.

Handling Script Execution

If you encounter permission errors while trying to execute a script, ensure that the script has the correct execute permission. You might need to use sudo if you don't have the necessary permissions. For example: sudo ./myscript.sh.

Additionally, you can run scripts without providing the explicit path by adding the script's location to your system's PATH variable.

For a more in-depth discussion of shell scripting, stay tuned for an upcoming blog where I'll delve into the details of this topic. 😄

12. Linux Man Pages: Your Comprehensive Guide

Navigating the Linux Man Pages

Linux Man pages, short for manual pages, provide detailed documentation for various commands, utilities, and functions available on your system. They offer a comprehensive guide to understanding command usage, options, and examples. Man pages are a valuable resource for both beginners and experienced users.

To access a man page, use the man command followed by the name of the command you want to learn about. For example, to view the man page for the ls command, you would type: man ls.

Man Page Sections

Man pages are organized into different sections, each focusing on a specific category of information. Here's a quick overview of the main sections:

• Section 1: User Commands

• Section 2: System Calls

• Section 3: Library Functions

• Section 4: Special Files (e.g., device files)

• Section 5: File Formats and Conventions

• Section 6: Games and Screensavers

• Section 7: Miscellaneous

Navigating Within a Man Page

When you access a man page, you'll find detailed information about the command, its options, and usage examples. Man pages are navigated using keyboard shortcuts:

• Press Space to move forward one page.

• Press Enter to move forward one line.

• Press B to move back one page.

• Press Q to quit the man page viewer

Getting Help from Man Pages

Man pages offer invaluable help in understanding commands and their usage. If you're unsure about how to use a specific command, simply consult its man page. For instance, to learn more about the grep command, enter: man grep.

Online Man Pages

While you can access man pages directly from your terminal, several online resources provide man pages as well.

Websites like (http://man7.org/linux/man-pages/) and (https://linux.die.net/man/) host extensive collections of man pages that you can search and browse.

Conclusion

In this comprehensive guide to Linux and Shell for Software Engineers, we've covered a wide range of topics essential for mastering Linux command-line usage and shell scripting. From understanding the Linux file system hierarchy and navigating directories to configuring networks, writing shell scripts, and exploring Linux Man pages, you now have a solid foundation to dive deeper into the world of Linux.

Remember that practice makes perfect. The more you work with Linux commands, the more comfortable and proficient you'll become. Experiment with various commands, create shell scripts for automation, and explore the vast capabilities Linux offers.

To further enhance your knowledge and skills, consider exploring these additional resources:

• Linux Documentation Project: A collection of guides, how-tos, and tutorials covering a wide range of Linux topics.

• Linux Command: A resourceful website providing tutorials and explanations of Linux commands.

• Linux Journey: An interactive learning platform offering lessons on Linux command-line usage and system administration.

• Bash Scripting Tutorial: A beginner-friendly tutorial on Bash scripting with practical examples.

For a more in-depth discussion about shell scripting, stay tuned for my upcoming blog where I'll delve into the details of this topic: Your Comprehensive Guide to Shell Scripting (Coming Soon).

With the knowledge gained from this guide and continuous exploration, you'll be well-equipped to excel as a Software Engineer working in Linux environments. Happy coding and exploring the world of Linux!

Thank you for reading! If you have any questions or feedback, feel free to reach out. Happy coding! 🚀💻

Git is a powerful version control system that allows developers to manage and track changes in their codebase efficiently. This guide will delve into essential Git commands and techniques to streamline your development process. From understanding commit history to managing branches and remote repositories, let's explore the key aspects of Git.

Note: If you're new to Git, the initial encounter might feel a bit perplexing. However, by following along and revisiting this guide multiple times, you'll gradually gain confidence and grasp the concepts thoroughly. I've organized the sections in an efficient order to enhance your comprehension. Rest assured that investing time in understanding Git will greatly benefit you.

Getting Started with Git Basics

Before diving into the more advanced Git commands, let's cover the fundamental concepts that form the foundation of version control.

Initializing a Git Repository

To start using Git for version control in your project, you need to create a Git repository. This can be done using the git init command. Navigate to your project's directory and execute the following command

git init

This command initializes an empty Git repository in the current directory. Once initialized, the directory will contain the necessary Git configuration and metadata to track changes in your project.

Staging and Committing Changes

Once you've made changes to your project files, you can use Git to track and manage those changes. The process involves two main steps: staging and committing.

Staging:

To stage changes for commit, use the git add command. This tells Git which changes you want to include in the next commit. For example:

git add file1.txt       # Stage a specific file
git add .               # Stage all changes in the directory

Checking Status:

You can check the status of your changes using the git status command. It provides information about which files are modified, staged, or untracked:

git status

Sample Output:

On branch master
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        modified:   file1.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        newfile.txt

Viewing Differences:

To see the differences between your working directory and the last commit, use the git diff command. This helps you review the changes before staging:

git diff               # Shows changes not yet staged
git diff --staged      # Shows changes in the staging area

Sample Output:

diff --git a/file1.txt b/file1.txt
index 1234567..89abcdef 100644
--- a/file1.txt
+++ b/file1.txt
@@ -1,4 +1,4 @@
-Hello, old content.
+Hello, new content.

Committing:

After staging your changes, you can commit them to the repository using the git commit command. This creates a snapshot of your changes with a corresponding message to describe the commit:

git commit -m "Added new feature"

Sample Output:

[master 1234567] Added new feature
 1 file changed, 1 insertion(+), 1 deletion(-)

Checkout the guidelines for commit messages here: https://gist.github.com/joshbuchea/6f47e86d2510bce28f8e7f42ae84c716

Viewing Commit History with Git Log

To view the commit history of your repository, you can use the git log command. This provides a chronological list of commits, including commit messages, authors, dates, and commit hashes:

Show previous commits

git log

Sample Output:

commit b5ec24aef8e74b9f56c0623a0efb6f81e0c3d9a9
Author: John Doe <john@example.com>
Date:   Wed Aug 18 15:24:47 2023 -0400

    Added new feature X

commit 8a73d5e9b9a5d14699ed82efcc2dfb82c1f2abf8
Author: Jane Smith <jane@example.com>
Date:   Tue Aug 17 10:12:36 2023 -0400

    Fixed issue #123
...

Show in a single line

git log --oneline

Sample Output:

b5ec24a Added new feature X
8a73d5e Fixed issue #123
...

Show the names of the committed files as well

git log --oneline --name-only

Sample Output:

b5ec24a Added new feature X
src/file1.js
src/file2.css

8a73d5e Fixed issue #123
src/file3.js

f9c7a21 Updated documentation
docs/readme.md
docs/api.md
...

Collaborating with Remote Repositories

Git facilitates effortless teamwork through remote repositories, allowing multiple developers to collaborate on the same project even if they are not in the same location. To establish this connection, you use a connection string, which is basically the URL of the remote repository. This URL typically ends with ".git" and serves as the pathway for your local repository to communicate with the remote one. This connection is a crucial step in ensuring that changes made by different team members can be shared and synchronized effectively.

Add remote git repo to existing local git repo

# git remote add [ALIAS] [CONNECTION_STRING]
git remote add origin https://github.com/yourusername/yourrepository.git

Clone Remote Repository as a new local repo

The git clone command is used to create a copy of a remote Git repository on your local machine. This allows you to start working with the repository, make changes, and synchronize with the remote repository as needed. Both SSH and HTTP URLs are supported.

git clone git@github.com:username/repository.git

Sample Output:

Cloning into 'repository'...
remote: Enumerating objects: 200, done.
remote: Counting objects: 100% (200/200), done.
remote: Compressing objects: 100% (150/150), done.
remote: Total 200 (delta 50), reused 180 (delta 30), pack-reused 0
Receiving objects: 100% (200/200), done.
Resolving deltas: 100% (50/50), done.

List all remote repositories linked to the current local git repo

git remote -v

Sample Output:

origin  https://github.com/yourusername/yourrepository.git (fetch)
origin  https://github.com/yourusername/yourrepository.git (push)

Pushing commits to the remote repo

Code is pushed to the remote repo to keep local and remote code in sync

# git push [REMOTE_ALIAS] [BRANCH_NAME]
git push origin new-branch

Navigating and Managing Branches

Imagine you're working on a big puzzle with your friends. Instead of messing up the whole puzzle while trying different pieces, you decide to make copies of the puzzle and try different pieces on each copy. These copies are like branches in Git. You can experiment on them without worrying about ruining the original puzzle.

When you're sure that the pieces you tried on one copy are correct, you can put those pieces on the original puzzle. This is like merging your changes from a branch back to the main work. Git helps you manage these copies and changes in a smart way so that everyone can work together on the puzzle without making a mess. Branching is a core concept in Git, enabling parallel development efforts.

List branches

git branch

Sample Output:

  master
* new-branch
  feature-branch
  ...

List all branches (local + remote)

git branch -a

Sample Output:

  master
* new-branch
  feature-branch
  remotes/origin/master
  remotes/origin/development
  ...

Create a new branch named new-branch

git branch new-branch

Switch to new-branch

git checkout new-branch

Create and switch to new-branch

git checkout -b new-branch

Create a new branch named new-branch based on the existing branch existing-branch and switch to new-branch

git checkout -b new-branch existing-branch

Delete new-branch

git branch -d new-branch

Visualize the commit history along with the branch they were committed on in graph form

git log --graph --decorate

Sample Output:

* commit 7a2c9f3 (HEAD -> master)
| Author: John Doe <john@example.com>
| Date:   Mon Aug 15 09:32:27 2023 -0400
| 
|     Fix critical bug in feature A
| 
* commit 43e8ab2
| Author: Jane Smith <jane@example.com>
| Date:   Fri Aug 12 17:55:13 2023 -0400
| 
|     Add new feature B
| 
| * commit 05b7f6c (feature-A)
| | Author: Alice Johnson <alice@example.com>
| | Date:   Wed Aug 10 11:21:09 2023 -0400
| | 
| |     Implement feature A
| | 
| * commit f1a2d8e
|/  Author: Bob Williams <bob@example.com>
|   Date:   Tue Aug 9 08:12:37 2023 -0400
|   
|       Initial commit
...

Merging Changes with Git Merge

Merging combines different lines of development. Git offers two merge modes:

• Fast-forward (--ff-only): Imagine you and a friend are working on the same document. You make a copy of the document, work on it separately, and your friend also makes changes to their copy. When your friend is done and you want to combine your changes, if they haven't made any new changes since you started, you can simply take their copy and add your changes to it. This is like a fast-forward merge. It's easy and quick because you're just adding your changes to the latest version.
• No Fast-forward: Now, let's say your friend made more changes to their copy after you started working. When you're done and want to combine your changes, it's a bit more complex. You need to carefully figure out where your changes fit in with their new changes. This is like a no fast-forward merge. Git helps you blend both sets of changes together, creating a new version that includes everyone's work, even if things got a bit more tangled.

So, in simple terms, fast-forward merge is like adding your changes to the latest version, and no fast-forward merge is like carefully combining changes when things have changed on both sides.

Merge new-branch into master

git checkout master
git merge new-branch

Pulling Changes with git pull: Combining Fetch and Merge

The git pull command plays a crucial role in keeping your local repository up-to-date with changes from a remote repository. It's essentially a combination of two separate steps: fetching changes and merging them into your current branch. This ensures that your local branch reflects the latest state of the remote repository.

Fetching Changes

When you run git pull, the first thing it does is perform a git fetch. This action retrieves all the new commits and references from the remote repository, effectively updating your local tracking branches to match the remote branches. This is an essential step as it helps you understand what changes have been made on the remote repository without automatically integrating them into your working branch.

Merging Changes

After the fetch, git pull takes an additional step: merging the fetched changes into your current branch. This is where the git merge part comes into play. If there are new commits on the remote branch you're tracking, git pull will attempt to automatically merge those changes into your local branch. If the merge can be completed automatically without conflicts, it will do so, effectively updating your local branch with the latest changes.

Handling Conflicts

In cases where there are conflicting changes between your local branch and the remote branch, git pull will pause and prompt you to resolve the conflicts manually. You'll need to edit the conflicting files, choose the desired changes, and then complete the merge using git commit.

Using git pull

The basic syntax for using git pull is as follows:

git pull <remote_name> <branch_name>

For instance, to pull changes from the remote repository named origin and integrate them into your current branch main, you would use:

git pull origin main

Sample Output:

From https://github.com/your-username/your-repository
 * branch            main       -> FETCH_HEAD
Updating abc1234..def5678
Fast-forward
 file.txt | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

By combining the actions of fetching and merging, git pull streamlines the process of updating your local repository with the latest changes from the remote repository. However, keep in mind that frequent use of git pull may lead to a complex history if not managed carefully. It's always a good practice to commit your local changes before pulling to ensure a smoother integration of updates.

Managing Revisions with Reset and Revert

Managing revisions involves undoing changes or resetting commits:

Revert a commit

git revert COMMIT_HASH

It creates a new commit that reverts all the changes of the given commit

Note: This will not delete the commit from the history

Reset a commit

Resetting a commit removes the commit from history and does not create a new commit.

Soft Reset

Using this option, we still have the changes in the working area

git reset --soft HEAD~1

To check the reset changes: git status

Hard Reset

This will reset the commit without saving any of the changes in the working area

git reset --hard HEAD~1

Restoring Files

To discard changes in specific files and restore them to their state in a specific commit, you can use the git restore command. This can be particularly useful when you only want to undo changes in specific files without affecting the entire commit:

git restore --source=<COMMIT_HASH> <FILE_PATH>

Sample Output:

Restored <FILE_PATH> from commit <COMMIT_HASH>

Stashing Changes for Later with Git Stash

Sometimes you're making changes to your code but suddenly need to switch to a different task or branch. Rather than committing incomplete changes, which can clutter your commit history, Git offers a handy solution called stash.

Git stash provides a way to temporarily store your working directory changes without committing them. This allows you to switch branches, work on a different task, or perform other operations without affecting your current changes. Once you're ready to return to the original task, you can apply or pop the stash back into your working directory.

The stash acts like a stack, where you can stash multiple sets of changes sequentially. This lets you manage different sets of changes across different tasks without any risk of losing your work.

Stash a change

git stash

Pop out the latest stash back to the working area

git stash pop

List all stashes

git stash list

Sample Output:

stash@{0}: On your-branch: b5ec24a Added new feature X
stash@{1}: On master: 8a73d5e Fixed issue #123

List the contents of a specific stash (say 1)

git stash show stash@{1}

Sample Output:

 src/file1.js  |  5 +++++
 src/file2.css | 10 +++++++++-
 2 files changed, 14 insertions(+), 1 deletion(-)

Pop a specific stash (say 1)

git stash pop stash@{1}

Sample Output:

On branch master
Changes from stash@{1} applied
Dropped refs/stash@{1} (8a73d5e9b9a5d14699ed82efcc2dfb82c1f2abf8)

Refining Your History with Git Rebase

Git rebase is a powerful command that enables you to rewrite commit history by incorporating changes from one branch onto another. Unlike merging, which creates new merge commits, rebasing results in a linear commit history, making it ideal for maintaining a clean and organized history.

How Git Rebase Works

When you perform a rebase, Git identifies the common ancestor of the two branches (the branch you're rebasing onto and the branch you're rebasing from). It then applies each commit from the branch you're rebasing onto, followed by the commits from the branch you're rebasing from, effectively placing your changes on top of the target branch.

Rebasing Process

• Choose the Base Branch: Start by checking out the branch you want to rebase onto. For example, if you're on the feature branch and want to rebase it onto master, you'd use:

  git checkout master
  

• Start the Rebase: Initiate the rebase with the git rebase command, followed by the name of the branch you want to rebase (e.g., feature):

  git rebase feature
  

• Resolve Conflicts: If conflicts arise during the rebase, Git will pause and allow you to resolve them. After resolving each conflict, use git add to mark the conflict as resolved and git rebase --continue to proceed.
• Finish the Rebase: Once all conflicts are resolved, Git will complete the rebase and apply the changes from the source branch onto the target branch.

  Sample Output:

  Applying: Add new feature
  Applying: Fix issue #123
  

Interactive Rebase

Git also offers an interactive mode for rebasing, which allows you to modify commits during the process. This is useful for squashing or splitting commits, reordering commits, and more. To initiate an interactive rebase, use:

git rebase -i <COMMIT_HASH>

In the interactive rebase, you can use the squash command to combine commits into one. This is particularly useful for cleaning up your commit history:

• Change pick to squash or s for the commits you want to squash.
• Save and close the editor.

For example, to squash the last two commits into one, your interactive rebase file would look like this:

pick abcdef1 First commit
squash 1234567 Second commit
squash 89abcdef Third commit

Use Cases for Rebase

• Feature Branch Integration: Rebase is useful when you want to incorporate the latest changes from the main branch into your feature branch before merging.
• Commit Cleanup: Interactive rebase allows you to squash or edit commits, creating a cleaner commit history.
• Maintaining a Linear History: If you want a linear commit history instead of a merge commit history, rebase is the way to go.

With Git rebase, you can shape your commit history to be more coherent and organized. However, remember that rewriting history should be done with caution, especially in a shared repository.

Selective Commit Integration with git cherry-pick

The git cherry-pick command allows you to pick specific commits from one branch and apply them to another. This can be helpful when you want to incorporate changes from a particular commit without merging or rebasing an entire branch. Here's how to use git cherry-pick:

git cherry-pick <COMMIT_HASH>

For example, if you want to apply the changes from commit abc123 onto your current branch, you would use:

git cherry-pick abc123

Sample Output:

[feature-branch 1234567] Fix issue #123
 1 file changed, 1 insertion(+), 1 deletion(-)

Reviewing Actions with Git Reflog

The git reflog command provides a comprehensive log of all actions taken in the repository:

git reflog

Sample Output:

b5ec24a HEAD@{0}: commit: Added new feature X
8a73d5e HEAD@{1}: commit: Fixed issue #123
...

Wait! Didn't we see a similar command at the beginning? The git log, right? Yes, I know these commands might appear very similar, but there's a substantial difference in their scope and focus. While git log primarily displays commit history, providing insight into changes and commit details, git reflog offers a broader perspective by recording all repository actions, including merges, rebases, resets, and more. It serves as a comprehensive timeline of actions taken within the repository, making it invaluable for troubleshooting, recovering lost commits, and understanding how various operations impact the repository's structure and history.

Conclusion

Congratulations! You've now explored essential Git commands and techniques that will enhance your development workflow. Whether you're managing branches, collaborating with remote repositories, or mastering the art of resetting and reverting, these skills will empower you to become a Git expert. Keep practicing and experimenting to unlock even more possibilities within the world of version control.

Check out these handy cheat sheets that you can refer to whenever you're working and need to look up a command:

• https://www.atlassian.com/git/tutorials/atlassian-git-cheatsheet
• https://docs.github.com/en/get-started/quickstart/git-cheatsheet
• https://education.github.com/git-cheat-sheet-education.pdf

Remember, Git is a versatile tool that plays a pivotal role in modern software development. By mastering these commands and techniques, you'll be better equipped to manage projects efficiently, collaborate seamlessly with teammates, and navigate the complexities of version control.

Happy coding! 💻 🚀